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Condltlons  are  given  for  the  existence  of  nonnegative 
solutions  of  the  equations 

1‘0’1*  - 

where  the  quantities  are  the  transition  probabilities, 
from  state  j  to  state  1,  of  a  transient  Markov  chain. 
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Tranelent  Markov  Chains 
wl*n  Stationary  Measures 

T.  £.  Harris 

1.  Introduction  and  Summary.  We  consider  Markov  chains 
x^,  n  -  0,  1,  •••,  wltn  denumerable  stater  denoted  by  Integers 
0,  1,  •••.  Let  Pjj  -  P^1  -  f>r(xn  +  1  -  J  I  xn  *  lj  and  let 
P^j11  denote  the  n-step  transition  probability  from.  1  to  J , 
n  ■  1 ,  2 ,  '  *  It  Is  assumed  throughout  that  for  each  1 ,  J 
there  Is  an  n  such  that  P^n  >  0*  For  other  terminology 
nee  Feller  [5],  Chapter  15*  (We  shall  refer  to  chains  satisfying 
the  above  assumption  as  Irreducible . ) 

We  shall  be  concerned  with  the  existence  of  solutions  of 
the  "equations  of  stat lonarity , " 

OD 

(1.1)  Q,  -  I  Q,P  ,  1  -  0,  1,  •  •  •, 

J-0  u  Jl 

In  the  cose  where  the  chain  Is  transient.  A  solution  will 
always  mean  a  set  ol  p os  1 1 1 ve  numbers  satisfying  (1.1). 

(It  1 8  readily  seen  that  If  a  set  o!  nonnegatlve  numbers, 
not  all  0,  satisfy  (l.l)  they  must  all  he  strictly  positive.) 

If  tne  chain  has  finite  mean  recurrence  times  It  Is  known 
that  there  Is  a  solution  whDee  elements  are  a  set  of 
probabilities,  -  1.  See  [5 j .  If  It  Is  assumed  only  that 

tne  chain  Is  recurrent,  Lerman  snowed,  ,  that  there  Is  a 
unique  (up  to  a  constant  multiplier)  solution,  wltn  LQ«  -  cl 

jL 

In  case  the  mean  recurrence  times  are  Infinite.  Leman  also 
shewed  by  examj  les  In  that  If  t:  e  on.- In  Is  transient 

t  lie  re  may  or  mav  not  l  e  solution?;. 
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In  this  note  we  obtain  a  necessary  condition  and  a 
sufficient  condition  for  the  existence  of  a  solution  of 
(l.l).  The  author  thinks  that  the  sufficient  condition  Is  in 
some  tense  close  to  being  necessary.  The  main  results  are  In 
Theorems  1  and  2.  The  corollary  to  Theorem  2  covers  a  number 
of  cases  of  Interest. 

2.  a  Necessary  Condition.  We  shall  use  the  following 
terminology ; 

Definition  1 .  A  path  from  Infinity  Is  a  sequence  of 
states  1 i,  1 2 ,  •  •  • ,  not  necessarily  all  distinct  but  con¬ 
taining  Infinitely  many  distinct  states,  such  that 


Pr(xn.l  •  !k  I  xn  '  Wl  >  °>  “  -  1.  2.  ••• 

Definition  2 .  A  simple  path  from  Infinity  Is  a  path 
from  Infinity,  all  oi  wh  se  states  are  distinct . 

Theorem  1.  In  order  that  (1.1)  should  have  a  solution  for 

an  Irreducible  transient  cnaln,  it  Is  necessary  that  there 

ex1 el  a  simple  path  from  Infinity . 

Proof.  Suppose  (l.l)  has  a  solution  Define  a  set 

of  "inverse  probabilities'*  by 

A  %/ 

(2.1,  p,.  -  Pl."  -  P.,n 

Since  Z  p  ,n  -  (Q,/Q.  )  Z  P  .n  <  oc  ,  tne  cnaln  defined  by  the 
n  l .  nji 

l  ,  .  Is  transient.  Let  y  ,  n  >  o,  be  tne  variables  of  such  a 
l  J  n  — 

chain.  3t  Is  evident  that  for  alaoef  every  sa»ple  sequence 

tne  quantities  p  ,  \  are  all  positive.  Moreover, 

y<yi  yiy2 
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because  of  the  transient  character  of  the  chain,  It  Is  true 
for  almost  every  sequence  that  no  state  Ip  visited  Infinitely 
often.  Therefore,  almost  every  sequence  contains  a  subsequence 

y ' ,  yT ,  ••*,  such  that  the  y?  are  all  dietlnct  and  p  ,  ,  >0. 

0  1  1  ylyi+l 

Heferrlng  to  (2.1)  we  3ee  that  there  Is  thus  a  simple  path 
from  Infinity.  This  completes  tne  proof  of  Theorem  1. 

As  an  example,  consider  the  renewal  process,  for  which 
Derman  showed  directly  that  there  Is  no  solution  in  the  transient 
case.  Here  p ^  -  0  unless  J  «  0  or  1  *1,  and  It  is  evident 
that  there  lr  no  simple  path  from  infinity.  In  the  renewal 
process  the  state  0  has  a  special  role;  every  path  from 
Infinity  must  contain  It  Infinitely  often.  However,  It  can 
be  shown  by  examples  that  even  if  there  Is  no  simple  path  from 
Infinity,  there  need  not  exist  any  such  distinguished  state. 

In  fact,  an  example,  which  we  do  not  give  here,  shows  that 
even  If  no  finite  set  of  states  has  the  property  that  every 
path  from  Infinity  Intersects  It  Infinitely  often,  there  need 
exist  no  simple  path  from  Infinity. 

3-  Conditions  on  the  Zeros.  The  condition  of  Theorem  1 
la  not  sufficient  fcr  the  existence  of  a  solution  to  (l.l). 

Note  that  this  Is  a  condition  on  tne  location  o:  the  zeros  In 
the  matrix  (Pj  ,).  The  following  two  examples  show  that  no 
condition  on  the  zeros  can  be  bctn  necessary  and  sufficient 
for  the  existence  ol  a  solution  In  the  transient  case,  since 
the  zeros  of  the  two  examples  are  in  tne  same  pi  ice  ana  one 


has  a  rolutlon  wille  trie  other  does  net. 


iWm 
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Example  1.  Take  *  1  -  ®  -  e  ,  P 


1,1-1 


—l  ^ 

e  ,  P  -  e  ,  1  -  1,  2,  PQl  -  1.  This  chain  Is  clearly 


transient  and  irreducible.  Theorem  2  below  can  be  applied  to 
snow  tnat  there  !s  a  solution  to  (l.l).  We  shall  not  give  the 
detai Is  . 


fcxanple  2.  *  Take  Pj  , ,  + 1  -  1  -  0)  .  h  ,  t_i  "  (t)  •  F 


lo 


(?)  -  (i)  • 


1  -  1,  2,  *•*;  Poi  =  1.  This  chain  Is  likewise 


transient  and  Irreducible.  We  now  pro/e  that  In  this  case 
(1.1)  has  no  solution. 

Proof  tnat  there  Is  no  solution  f or  Example  2.  Suppose 
that  a  solution  j  exlots.  First,  the  cannot  be  bounded 


For 

1 

setting  1 

*  0  In 

(1. 

1), 

we  have 

00 

CD 

p  n 

JO  ' 

(3- 

1) 

• 

I  Q.P 

J-o  J 

JO 

m 

*  QJ 

J-o  J 

OD 

N 

P  n 

JO  ’ 

N«o  " 

J-o 

Z 

n-1 

Qj 

N-1, 

If 

the  Qj  were 

bound  t 

y  k 

we 

would 

have 

OD 

N 

OD  OD 

(3. 

2) 

NQ  < 

vJ  — 

K  Z 

J-o 

n 

Z 

-1 

K  Z  Z 

J-o  n-1 

n 


Jo 


New  1^®^  P  ,Qn  Is  tr.e  expected  total  number  of  visits  to  state  0 


of  a  ^article  starting  In  state  j  and  Is  equal  to  L  multiplied 

J  ^ 


by  t;.e  expected  number  of  visits  to  0  starting  In  C,  where  L, 


Is  toe  probability  of  reaching  0  from  j  at  least  once.  It 
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io  readily  seen  that  H, L.  <  oo  ,  which  means  that  (3*2) 

J  J  w 

cannot  hola  for  all  N.  Hence  the  ,  If  they  exist,  must  be 
unbounded.  Next,  observe  that  equations  (1.1)  hav6  the  form 
In  this  example 


{yi)  %  -  £3  [(?)  -  (5)  J  V  Q1  ■  'o  +  «2' 

(J.-)  Qt  -  [1  -  (5)  ]  «i-i  +  ©  <W  1  *  2'  '•  •••  • 


Since  1 


-  (fj  *  (t/+  <  1  for  1  2  2. 

cannot  be  as  large  aa  Max  (< 


we  see  fro*  (3.4)  that 
i  -  2,  3#  ••••  Ilnoe 


the  Q.  are  unbounded  they  must  therefore  ultimately  Increase 


monotonlcal ly  to  Infinity.  Hence,  there  is  an  1  such  that  If 
J  >  1,  Qj  >  Tr*«n,  from  (3.4) 


(3-5) 


> 


1-1 


(i/<0J+1 


-  J  -  i’  »♦!.••• 

By  repeated  application  of  (3.^)  we  have,  after  a  little 

2 

simplification,  >  2  ^l-l'  n  ■  1»  2,  •••  This  Implies 

that  the  sum  In  (3.3)  Is  infinite,  a  contradiction.  Hence  (l.l) 
can  have  no  solution. 


■* .  A  Sufficient  Condition. 

Definitions.  Let  P  *  -  P  , ,  and  In  general 

"  *  w  A.  v 

(4.1)  kPljn  m  Fr  ixn  m  xm  *  k,  1  <;  m  <  n  |  xQ  -  lj  , 


i 

r 
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n, 


(4,?)  Lk1^j)  "  rfj  llkr  ‘ rl  '  *ki 


♦  P, 


"  L,  Pr  lxn  ’  1;  xm  ♦  *•  1  <  m  <  n;  xn— 1  >  J  I  x0  "  kl 
n-l 


Lkl(j)  is,  If  k  >  J,  the  probability  tluit  a  partlclt,  starting 
at  k,  will  retch  1,  the  first  visit  bslng  Immediately  preceded  by 
a  state  with  lndsx  >  J.  As  before,  -  L^(0)  Mill  denote 

the  probability  of  reaching  1  at  all,  having  started  from  k. 

Tneorem  ? .  In  order  that  ( 1 . 1 )  should  have  a  solution  for 
an  Irreducible  transient  chain,  the  following  condition  Is 
sufficient:  there  exists  an  Infinite  set  K  of  states  such  that 


(♦•3) 


11m 

J  — >  OD  , 
k  — >  qd  ,  kCK 


Lkl(j)/L,;l  -  0, 


1-0,  1,  2, 


Corollary.  If  the  chain  Is  transient  and  Irreducible,  and  If 
for  each  1,  ■  0  except  for  a  finite  set  of  values  of  k, 

then  (1.1)  has  a  solution. 

Proof  of  Theorem  2.  In  the  recurrent  case  Chung,  [2\  , 
8hOMed  that  we  can  pick  an  arbitrary  state,  say  0,  and  define 


ao 


n 


-  L  0PQk  i  obtaining  a  solution.  Here  Is  the  expected 
n-l 

number  of  visits  to  k  between  visits  to  0,  and  we  have  (note 

that  because  of  recurrence  Q’  -  1) 

o 


(‘*.4) 


ob  ,  ao  ac  ,  1 

L  P  «n  -  P  ,  ♦  L  L  P  ,  P.,  -  P  «  ♦ 

0  ol  o  ol  .  ^  ,  o  oj  J1  0  ol 


n-l 


n-2  >1 


oo 


1  QjP  , 
J-l  J  J 


oo 

I  Q’.P 


J-o 


J  J. 


since  ■  Pqi*  ■  0^PQl**  *n  tran8ient  case  this  does 

not  give  a  solution.  It  seems  reasons.^**  to  try  Instead 


CD 

I  P. 


n 


QJ  -  -  -ki  3 

J-l  K1 

the  expected  total  number  of  visits  to  1,  starting  at  k. 

It  Is  perhaps  better  to  think  of  as  the  average  density  of 
particles  at  1,  If  there  Is  a  source  putting  one  particle  per 
time  unit  Into  the  system  at  k.  Although  Q”  Is  not  Itself  a 
solution  we  may  hope  that  It  approaches  one  as  the  "source" 
k  moves  off  to  Infinity.  We  must  also  normalize  to  keep  the 
within  bounds.  Hence  we  define 


(*•5) 


ao  os 

Srt-  S  Pkl  /  S  r  . 

n-1  kl  n-1 


l.k  -  0,  1, 


We  shall  show  that  if  the  conditions  of  Theorem  2  hold,  then 
there  Isa  sequence  (k^  \  such  that 

(4.6)  11m  Q.  .  -  Q.  ,  1  -  0,  1,  ..., 

m  — >ao  m ' 1  1 

where  the  are  a  solution  of  (l.l). 

Definition?.  Let  6^,  1  f  J,  be  the  probability  that  the 
state,  given  to  be  Initially  1,  reaches  J,  before  reaching  1; 
let  1  -  1  4  J,  be  the  probability  that  the  state,  Initially 

1 ,  returns  to  1  without  reaching  j . 

In  general,  6^ ,  4  4jj  for  transient  chains. 

Now  suppose  tne  state  Is  Initially  1  4  J#  ai’d  let  V 
be  the  total  number  of  visits  tc  J  which  precede  any  further 
visit  tc  1.  Then  clearly 
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CD 


(4.7)  E(V)  - 
Hence  m«  have 

(“•8>  \j  -  \i*u  /  ♦ji ♦  nfx  ipkjn 
Then 

(4.9; 


1  +  3 


CD  CD  00  - 

Si  -  2,  Si  •  pkl  *  1  1  Sr  Prl 

K  n-1  K1  n-2  r-0  Kr  ri 


kl 


J-l 

♦  z 

r*»0 

SrPrl 

CD 

r 

®lr 

00  -1 

♦  z 

r-J 

[Si 

^rl 

♦*  pP“r  i 

n»’l  J 

prl’  J  >  »• 

of  (4. 

9)  by 

Sco  * 

and  recalling 

(4.2),  *»• 

obtain 


(4.10) 


S<1  J-l  /Sir  \  "Tel  ao 

-  -  Z  I -  P_,  +  -  Z 


Scl 


Sco  r-°  \  See 


rl 


Sco  r"^  wrl 


rl 


j  >  1. 


From  (4.5)  Me  aee  that  Interchanging  1 

and  J  glvee 


eij  /  ^ji  <  S<j  /  Si  <  ^ij  /  ejr 

Hence  ratios  such  as  QJ(r  /  Qk0  are  bounded  away  from  0  and  ac 
by  numbers  which  may  depend  on  r  but  are  Independent  of  k. 
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Also,  -  L^(l  ♦  Q^).  Thus,  the  condition  of  Theorem  2 

Implies  that  If  k  E  K,  k  -4  oo  ,  J  — >  ao  ,  then  L^U)  /  0  — *  0 

for  each  1.  The  above  remarks  also  Imply  that  there  Is  a 

sequence  <k  l  ,  m  -  1,  2,  . ..,  with  k  f,K,  such  that 

i  m  «  m 

v 

Urn  -  -  Q1#  say, 

0 

exists  for  each  1.  Since  for  each  1  the  ratio 
Is  bounded,  and  since  * 

J  'eir  /  ♦rl*  Pri 

Is  arbitrarily  small  for  J  sufficiently  large,  the  limits  ^ 
must  satisfy  (l.l).  This  concludes  the  proof  of  Theorem  2. 

5-  Remarks .  Blackwell,  [lj  was  concerned  with  the 
number  of  bounded  solutions  for  transient  chains  of  the  system 

(5a)  <1  •  Jo  PoV 

There  are  some  connections  between  solutions  of  (l.l)  for  a 
given  chain  and  solutions  of  (5.1)  for  the  Inverse  chain. 
However,  the  author  has  not  obtained  any  definitive  results  In 
this  direction. 

We  leave  unsettled  the  determination  of  conditions  Implying 
that  (l.l)  nas  only  one  solution  (up  to  a  constant  multiplier). 
Is  clear  that  there  Is  some  flexibility  in  the  cnclc*  of  the 
sequence  k^  ol  section  4  and  In  some  cases  dliferent  sequences 
obviously  lead  to  different  solutions. 
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